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* NOTICES * 

iTPO and NCIPI are not responsible for any 
damages caused by the use of this translation. 

1 .This document has been translated by computer. So the translation may not reflect the original precisely. 

2 **** s h ows WO rd which can not be translated. 
3. In the drawings, any words are not translated. 



CLAIMS 



[Claim(s)] 

[Claim 1] The image experience system characterized by to have the player positioning means which is an 
image experience system for the game which advances by arranging an item on a game board, and searches for 
the positional information of a player view, a generation means generate the computer graphics according to the 
positional information of the view of said player according to the item on a game board, and the head mount 
display which is made to superimpose said generated computer graphics on the image of the real world, and is 
displayed on it. 

[Claim 2] The positional information of said player is an image experience system according to claim 1 
characterized by being the information which shows the relative location from said game board of said player 
view. 

[Claim 3] Furthermore, it is the image experience system according to claim 1 which has a measurement means 
to measure the attitude information of said player, and is characterized by said player positioning means 
searching for the positional information of said player view from the attitude information of said player, and the 
position information proofread beforehand. 

[Claim 4] Furthermore, it is the image experience system according to claim 1 which has the camera fixed to 
said head mount display, and is characterized by searching for the positional information of said player view 
when said player positioning means analyzes and carries out the image recognition of the photography image of 
said camera. 

[Claim 5] It has the position sensor which measures the location of a player, and the camera fixed to said head 
mount display. Furthermore, said player positioning means The first spotting section which searches for the 
positional information of a player view from the output from said position sensor, The image experience system 
according to claim 1 characterized by having the second spotting section which searches for the positional 
information of a player view from the photography image of said camera, and searching for the positional 
information of the view of said player based on the said first and second [ said ] reliability of each output value 
of the spotting section. 

[Claim 6] It has the attitude sensor which measures the posture of a player, and the camera fixed to said head 
mount display. Furthermore, said player positioning means The first spotting section which searches for the 
positional information of a player view from the output from said attitude sensor, The image experience system 
according to claim 1 characterized by having the second spotting section which searches for the positional 
information of a player view from the photography image of said camera, and searching for the positional 
information of the view of said player based on the said first and second [ said ] reliability of each output value 
of the spotting section. 

[Claim 7] Furthermore, the image experience system according to claim 5 or 6 characterized by calculating 
correction value from the output value of said first spotting section or said 2nd spotting section, and amending 
the output value of said first spotting section or said 2nd spotting section using said correction value. 
[Claim 8] Furthermore, the image experience system according to claim 1 characterized by having an item 
actuation recognition means to recognize change of the item on said board game. 
[Claim 9] Said coma actuation recognition means is an image experience system according to claim 8 
characterized by recognizing the special mark identifier attached to the item. 

[Claim 10] The image experience system according to claim 9 characterized by using a visible or invisible bar 
code as a special mark identifier. 
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[Claim 1 1] The image experience system according to claim 9 characterized by using a RFID transponder as a 
special mark identifier. 

[Claim 12] Said item actuation recognition means is an image experience system according to claim 8 
characterized by recognizing the configuration of an item, a pattern that it was drawn on the item, or its both, 
using image recognition. 

[Claim 13] The image experience system according to claim 12 characterized by using the photography image 
of the camera fixed to said head mount display. 

[Claim 14] Said item actuation recognition means is an image experience system according to claim 12 
characterized by recognizing the item changed by recognizing that the item has been arranged in the specific 
location of a specific camera. 

[Claim 15] The image experience system according to claim 14 characterized by constituting the guide make an 
item easy to arrange from computer graphics, and making it display on the head mount display of said player in 
case a coma is arranged in said specific location of said specific camera. 

[Claim 1 6] The image experience system according to claim 1 characterized by for two or more players playing 
on one game board, and expressing the result of the combined control by two or more players to each head 
mount display as each player view. 

[Claim 17] The information processing approach characterized by being the information processing approach 
for the game which advances by arranging an item on a game board, inputting the positional information of a 
player view, generating the computer graphics according to the positional information of the view of said player 
according to the item on a game board, making the image of the real world superimpose said generated 
computer graphics on the head mount display with which the player has equipped, and making it display. 
[Claim 18] The positional information of the view of said player is the information processing approach 
according to claim 1 7 characterized by being the information which shows the relative location from said game 
board of a player view. 

[Claim 19] The program characterized by controlling an information processor and performing information 
processing indicated by claim 17. 

[Claim 20] The record medium characterized by recording the program indicated by claim 19. 



[Translation done.] 
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DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[Field of the Invention] It is related with the game which advances by arranging an item on a game board. 
[0002] 

[Description of the Prior Art] There is a thing called a board game as a genre of a game from before. It is the 
game of the kind which advances a game, placing a coma, removing and moving [ this has the field classified 
on the board, and ] it on the field. For example, as a thing using the object which usually carried out the specific 
solid configuration as a coma, chess, a checker, backgammon and the game of go, shogi, sugoroku, etc. are 
known well. Moreover, there is also a thing using a card as a coma and this is called the card game, the thing 
using cards — Baba omission and Napoleon — it arranges seven and poker, blackjack, etc. are too many to 
mention. 

[0003] There are some which gave the specific role to each card in a card game using a card still more peculiar 
to a game. The game called the so-called card battle corresponds to this. In addition, although a special board 
may not be used since the field where many of card games were classified is simple, this has a board with the 
invisible field where it was classified, and it is possible that the player is recognizing and sharing the existence 
mutually. 

[0004] Such a board game and the card game itself assume a certain occurrence, and the item (for example, 
coma) also assumes the specific animal, the person, etc. in many cases. Moreover, the form was decided 
beforehand and neither a board nor a coma was those from which a pattern that it is drawn also changes 
according to the advance situation of a game. 

[0005] On the other hand, the game using MR (Mixed Reality) technique also exists. This builds the 
environment where a game is performed, by actual sets, such as stage setting and stage properties, and plays by 
a player entering into it. many cases — each player — see-through one — CG image doubled with game advance 
is superimposed on the image which is in sight when equipping with HMD (Head Mounted Display) and not 
equipping with HMD, and it displays on HMD. 
[0006] 

[Problem(s) to be Solved by the Invention] As for the board game of the above-mentioned conventional 
example, the form or pattern of a coma do not change according to the situation of a game. For example, even if 
it hits a card with which a battle scene is not said that a battle actually breaks out before it, either, and is referred 
to as "An angel issues - and directions", an angel does not necessarily actually talk. 

[0007] Therefore, since there was no corresponding display in spite of assuming a certain scene, the game itself 
had the problem that presence was inferior. Moreover, there was a problem that it was difficult to grasp the 
advance situation of a game at a glance by the same reason. 

[0008] On the other hand, although there is presence very much, installation of a game environment is very 
serious in MR game of the above-mentioned conventional example. Although the activity itself which creates a 
set tended to become large-scale, in addition to it, the location of the object in a set whenever it constitutes a set 
had to be measured. Moreover, modification of the contents of a game was difficult. 

[0009] This invention is made in view of an above-mentioned point, and is boiled in the fun of an actual board 
game, in addition presence aims [ this invention ] at increase and making the advance situation of a game 
further easy to grasp. 

[0010] And as compared with the game using the conventional MR technique, installation is easy and it aims at 
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enabling it to correspond also to modification of the contents of a game comparatively flexibly. 
[0011] 

[Means for Solving the Problem] This invention is characterized by having the following configurations, in 
order to attain the above-mentioned purpose. 

[0012] Invention given in this application claim 1 is an image experience system for the game which advances 
by arranging an item on a game board, and is characterized by to have the player positioning means which 
searches for the positional information of a player view, a generation means generate the computer graphics 
according to the positional information of the view of said player according to the item on a game board, and 
the head mount display which is made to superimpose said generated computer graphics on the image of the 
real world, and displays on it. 

[0013] Invention given in this application claim 17 is the information-processing approach for the game which 
advances by arranging an item on a game board, and is characterized by to input the positional information of a 
player view, to generate the computer graphics according to the positional information of the view of said 
player according to the item on a game board, to make the image of the real world superimpose said generated 
computer graphics on the head mount display with which the player has equipped, and to make it display. 
[0014] 

[Embodiment of the Invention] Hereafter, the image experience system of this invention is explained based on a 
drawing. In addition, in each drawing, the same number is added about the same configuration. 
[0015] the image experience system explained below ~ a player ~ see-through one — I have HMD (head mount 
display) covered, and CG (computer graphics) is superimposed and expressed in the situation of performing the 
board game and the card game actually as a limited place called a game board. CG changes here according to 
the advance situation of a game. For example, if it is chess, a night is CG of the knight who rode on the horse, 
and when moving, it will serve as CG which a horse runs. Moreover, when taking a partner's coma, it fights 
with CG corresponding to a partner's coma, and is set to winning CG. 

[0016] According to this image experience system, it is alike in the fun of an actual board game, in addition 
presence increases, and it further becomes easy to grasp the advance situation of a game. 
[0017] Moreover, as compared with the game using the conventional MR technique, installation is easy and it 
can respond also to modification of the contents of a game comparatively flexibly. 

[0018] (1 st operation gestalt) Drawing 1 is drawing showing the example of the image experience structure of a 
system of the 1 st operation gestalt. 

[001 9] There is a game board 101 used as the place of a game, a player places a coma on a board 1 01 , or 
removes it, or it is made to move and it advances a game, a player — see-through one — a game is performed, 
with HMD 103 covered. It is fixed to HMD 103 and a position sensor 104 detects the position and posture of a 
player view. 

[0020] The "location" used by this proposal here, the "position", and the "posture" are defined. Both a 
"position" and a "posture" are included with a "location." That is, if it is called "positional information", it will 
be the information on both "position information" and "attitude information." A "position" is the information for 
directing one point in a specific space-coordinates system, and, in the case of a XYZ rectangular coordinate 
system, it is expressed with the group of three values of x, and (y, z). Moreover, if the event on the earth is 
expressed, it can also express with the group of three values, the LAT, LONG, and altitude (or depth), a 
"posture" can show which direction has been turned to from one point shown by the "position", and which 
rotates it around each axis of coordinates, or it carries out and, in the case of a XYZ rectangular coordinate 
system, can express [ the look determines each axis of coordinates and the include angle to make which can also 
use the position of one point of the arbitration on the direction turned to, or the regular direction of a look (for 
example, - Z direction), and / it can come out and ]. 

[0021] make it any — when there is especially no constraint, a "position" is a degree of freedom 3 and a 
"posture" is a degree of freedom 3. 

[0022] A position sensor is the sensor which can acquire the value of a "position" and six degrees of freedom of 
a "posture." 

[0023] Various things as a position sensor are sold and there are a thing using a magnetic field, a thing which 
photos and carries out the image processing of the marker with the camera installed outside, a thing which 
combined the gyroscope sensor and the acceleration sensor. 
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[0024] Moreover, the location of the head of a player hardly changes on the character of a board game, and 
during a play. Then, the sensor which measures only attitude information, i.e., an attitude sensor, can also be 
used using the value proofread as position information at the time of game initiation. In this case, although the 
inside of a game is measuring only three degrees of freedom of attitude information, it can be combined with 
the position information first proofread as a system, and can be processed as what has returned the value of six 
degrees of freedom. That is, it can be regarded as a location sensor by the attitude sensor and the proofread 
position data. 

[0025] The approach of the proofreading before a game is shown in drawing 2 . On the game board 101 , the 
marker 201 for discernment is attached in the four corners. Since a marker 201 is easy here, it supposes that it is 
arranged at the square and square die length of one side is 1. Although a player is located before a board, the 
position of the view 202 of a player is d from the front side at the front of a board. When a player looks at a 
board in this condition, compared with the front side, as for a board, the back side is observed short. Although 
height h from the 101st page of the board of a view 202 will depend the die length of the side of the observed 
this side also on the projection approach if the die length of the side of ml and the back is set to m2, it asks as 
follows. 
[0026] 

I/O of an image with h=(m22(d+l) 2 ml2d2) (/)0.5(ml2 m22) HMD 103 and the positional information from a 
position sensor 104 are processed by a game console or PCI 02. 

[0027] drawing 3 — see-through one — it is drawing having shown the internal configuration of HMD 103. There 
are a video see-through type and an optical see-through type of see-through HMD(s). 

[0028] In a video see-through type case, the light of the direct external world does not reach the eye of a player. 
The light from the external world changes the direction of a beam of light by the double-sided mirror 301 , and 
goes into an image sensor 302. Moreover, the image shown to a player is displayed on a display device 303, and 
goes into the eye of a player via the double-sided mirror 301 . Although it will become mere glasses if the output 
image of an image sensor 302 is inputted into the direct presentation component 303, CG which added and 
generated processing with the game console or PC 102 in the meantime can be indicated by superposition. 
[0029] In an optical see-through type case, while the light of the external world reaches the eye of a direct 
player, it seems that a superposition indication of CG was given to a player by displaying CG generated 
separately on coincidence. 

[0030] The light from the external world runs through a half mirror as it is, and goes into the eye of a player. It 
is reflected by the half mirror and the image displayed on coincidence by the display device goes into the eye of 
a player. In this case, although an image sensor is unnecessary, it is needed in using the image in a player view 
for an image processing. Or it is also possible to form a camera in image processings separately and to fix to 
HMD 103, without using an image sensor 302. 

[0031] A game console or PC 102 manages and runs a game like the usual game. 

[0032] In addition, in the case of a board game, it is necessary to proofread neither the set by stage setting and 
stage properties, nor the sensor with which the body is equipped as positional information at every installation 
that what is necessary is just to understand even the relative location of a board 101 and HMD 103. 
[0033] According to this operation gestalt, compared with MR game, a required set is compact. The installation 
which also includes proofreading with a pan is easy. The detail about MR game is indicated by the framework 
for the "compound actual mold amusement of the 22 A: compound sense of reality of for example, the 4th time 
convention collected works of the Virtual Reality Society of Japan, and mounting (Design and Implementation 
for MR Amusement Systems). 

[0034] Moreover, according to this operation gestalt, presence can be raised compared with the conventional 
board game. 

[0035] Drawing 4 is an example of the image experience structure of a system to which this operation gestalt is 
applied. 

[0036] A game console or PC 102 manages advance of a game by the game Management Department 401. And 
according to advance of a game, CG (computer graphics) corresponding to each scene is generated in CG 
generation section 405. CG generation section 405 acquires the positional information of a player view from the 
player positioning means 402, in order to generate CG image seen from the player view. The information is 
analyzed with a position sensor 104, and the position-sensor information processing section 403 which 
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determines the positional information of a player view is contained in the player positioning means 402. 
[0037] The position-sensor information processing section 403 performs format conversion of the data obtained 
from a position sensor 104, conversion to the system of coordinates used by the system, amendment of the 
difference of the location and HMD 103 view that the position sensor is attached, etc. 

[0038] Since CG generated in CG generation section 405 is the image seen from the player view, in video see- 
through type HMD, it superimposes on the image obtained from the image pick-up section 302 of HMD 103 in 
the image composition section 404, and it is displayed on it at a display 303. 

[0039] Since there is no need for image composition in optical see-through type HMD, the image pick-up 
section 302 and the image composition section 404 are unnecessary, and display the output of CG generation 
section 405 on a display 303 as it is. 

[0040] The information about the game itself and the so-called Ruhr are stored, and the game Management 
Department 401 holds a condition and a scene current in under game advance, and opts for and manages to 
which condition next it shifts. In the scene furthermore shown to a player using CG, drawing directions of CG 
are published to CG generation section 405. 

[0041] CG generation section 405 is arranged according to the directions from the game Management 
Department 401 to the world which is the internal representation of the virtual world to which the player is 
playing the model data which are the internal representation corresponding to each character. After carrying out 
internal representation also of model data and the world by the technique called a scene graph and generating 
the scene graph of a world, the rendering of the scene graph is carried out. The rendering of the scene which 
looked at the world from the location given from the player positioning means 402 is carried out in that case. 
[0042] A rendering may be performed on the memory for the display which may be performed on the memory 
of the interior which is not displayed and is called a frame buffer. Here, since it is easy, a rendering shall be 
carried out on the memory of the interior which is not displayed. 

[0043] In the image composition section 404, CG which CG generation section 405 generated is superimposed 
on the image picturized by the image pick-up section 302 in HMD 103. In order to indicate the image by 
superposition, the technique of alpha blending can be used. When it has Opacity A (alpha value) (0<=A<=1) as 
a pixel output format of the image composition section 404 in addition to the reinforcement of RGB called 
RGBA in three primary colors, it compounds with the value of Opacity A. 

[0044] For example, the value of (R2, G2, B-2), and opacity is set to A for the value of the pixel to which (Rl, 
Gl , Bl), and the image composition section 404 correspond the value of a certain pixel on the output from the 
image pick-up section 302 with a RGB value with a RGB value. 

[0045] The value of the corresponding pixel outputted to a display 303 is as follows then. 
[0046] 

(Rl*(l-A) +R2*A, G1*(1A) +G2*A, B1*(1A) +B-2*A) Although it is also possible to perform processing of 
this alpha blending within CG generation section 405, in order to explain a function plainly here, it has 
illustrated as a separate component. 

[0047] By making it the above configurations, a player is boiled in the fun of a game, in addition senses a rise 
of presence. Since CG on which it is furthermore superimposed is what synchronized with advance of a game, it 
becomes easy to grasp the advance situation of a game. 

[0048] Moreover, it is necessary to proofread neither the set by stage setting and stage properties, nor the sensor 
with which the body is equipped as positional information at every installation that what is necessary is just to 
understand even the relative location of a board 101 and HMD 103. 

[0049] In addition, this invention is not what was restricted to the game, and can be applied also to applications 
of various fields, such as a presentation, education, simulation, and visualization. 

[0050] (2nd operation gestalt) Drawing 5 is drawing showing the example of the image experience structure of 
a system of the 2nd operation gestalt. Only the configurations of the player positioning means 402 differ 
compared with drawing 4 . 

[0051] The player positioning means 402 consists of the cameras 501 and the board image recognition sections 
502 which were fixed to HMD 103. The image of the board 101 reflected in the image photoed with the camera 
501 changes with locations of the player view 202. Then, the image photoed with the camera 501 can be 
analyzed in the board image recognition section 502, and the location of a player view and a posture can be 
determined. 
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[0052] Although the image of a board 101 is recognized in the board image recognition section, when the 
marker 201 is attached to the game board 101, the image obtained with a camera 501 becomes the bent thing. 
The location of a camera 501 is determined based on this distortion. If correspondence of at least four points can 
be taken about a marker 201 , it is known that the location of a camera 501 can be determined. In this way, the 
location of the obtained camera 501 is amended based on the difference between a camera location and a player 
view location, the location of the player view 202 is computed, and it considers as an output. 
[0053] In addition, when the image pick-up section 302 is attached to HMD 103 beforehand, the image pick-up 
section 302 may be used instead of a camera 501, and there is equivalent effectiveness. 
[0054] according to this operation gestalt — a player positioning means — see-through one ~ since it is 
constituted by the camera fixed to HMD, and the board image recognition section and the board image 
recognition section determines the relative location of a board and a player view from the image of the board 
picturized with the camera, the positional information of a player view can be searched for without using a 
position sensor, and a configuration can be simplified. 

[0055] (3rd operation gestalt) Drawing 6 is drawing showing the example of the image experience structure of a 
system of the 3rd operation gestalt. 

[0056] Only the configurations of the player positioning means 402 differ compared with drawing 4 and 
drawing 5 . 

[0057] It has both components of the player positioning means 402 in drawing 4 and drawing 5 as a component, 
and has maximum ************ gQl further. 

[0058] Generally, the output of the location sensor 104 is weak to disturbance, and its stability is also low. 
Then, although what is necessary is just to use the positional information from the board image recognition 
section 502 as an output of a player positioning means, it does not restrict that the board 101 is not necessarily 
contained to the image pick-up range of a camera, but there are also elements which bar recognition, such as a 
hand of a player. In such a case, the reliability of the value of positional information falls. 
[0059] Then, it restricts in that case and the information from a position sensor 104 is used. Consequently, the 
period when the board 101 is recognized can acquire positional information with a high precision, without a 
positional information output breaking off. 

[0060] The UML activity Fig. showing processing of the maximum ************ i s shown in drawing 7 . 
[0061] First, it waits for the positional information data from the board image recognition section 502 from the 
position-sensor information processing section 403. In the place which had complete set of data of both, it 
judges whether it is what has suitable positional information data from the board image recognition section 502. 
If it is judged that it is suitable, when that is not right, the information from the position-sensor information 
processing section 403 will be used, using the information from the board image recognition section 502 as an 
output of the player positioning means 402. 

[0062] In addition, in such a case, this invention is not restricted, although it is explaining on the assumption 
that a value with it is acquired while being able to recognize normally from the board image recognition section 
502 here. [ unsuitable when a highly precise value is not so ] Although a value is computable depending on the 
image from a camera 501 , its reliability may be low, and if a position sensor of a certain kind is used, a very 
highly precise value will be acquired in the specific range, but if it separates from the range, reliability may fall 
gradually. It is required to design so that the value considered to be the optimal may be outputted based on the 
reliability over each output value of the position-sensor information processing section 403 and the board image 
recognition section 502. 

[0063] Even when either of the positional information searched for by two or more approaches is unsuitable 
according to this operation gestalt, reliable positional information can be searched for. 

[0064] (4th operation gestalt) Although it is in the method of a position sensor partly, there is a problem that the 
acquired value swings, by many methods. Although the axis of abscissa was set as time amount progress and 
the difference dV of the measured value and an original value was graph-ized, an example is shown in drawing 
8 . Although there are what swings small like a dotted line, and a thing which changes greatly like a continuous 
line, it is aimed at the case where the value changes greatly like a continuous line, with this operation gestalt. 
[0065] If it samples by sufficient frequency in big transition, the difference of two samples of continuous dV is 
minute. Then, when the value of the board image recognition section 502 is proper, after assuming it to be an 
original value, dV is calculated and it is correction value of position-sensor information. - dV is used. 
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[0066] Since the positional information to which it will start as 0 and the player positioning means 402 outputs 
the time of the value of the board image recognition section 502 becoming inaccurate [ transition of a big 
value ] by carrying out like this at this time serves as a continuous value, the displeasure by CG drawing 
location shifting suddenly for a player is lost. 

[0067] Since the change of dV by big transition is minute when the period when the value of the board image 
recognition section 502 is still more unjust is short enough, also when the value of the board image recognition 
section 502 turns into a value of the player positioning means 402 again, CG drawing location does not become 
discontinuity. 

[0068] The example of the image experience structure of a system of the 4th operation gestalt is shown in 
drawing 9 . 

[0069] Although the fundamental configuration has not changed compared with drawing 6 , in addition to 
maximum ************ goi, the output of the board image recognition section 502 is inputted also into the 
position-sensor information processing section 403, and it differs in that the output of the position-sensor 
information processing section 403 is inputted also into the board image recognition section 502 in addition to 
maximum ************ 601. However, the output of the position-sensor information processing section 403 
explains here as what can be disregarded about the part inputted into the board image recognition section 502. 
[0070] The UML activity Fig. showing processing of the position-sensor information processing section 403 is 
shown in drawing 10 . 

[0071] It usually passes along the information from a position sensor 104, it is processed, and positional 
information is calculated. The value is evacuated to Variable LastSensorPos. Variable LastSensorPos is an 
object variable and is referred to also from another thread described just in the back. 

[0072] Then, it evacuates to the calculated positional information temporarily by making into a return value the 
value which applied correction value. In addition, correction value is also an object variable and a value is set 
up by another thread described just in the back. Although a return value is a variable, this is only temporarily 
used, in order to perform exclusive activation. And finally the return ground is returned as an output of the 
position-sensor information processing section 403. 

[0073] Now, although it is necessary to calculate above-mentioned correction value, this is updated at any time, 
when there is an output from the board image recognition section 502. 

[0074] First, it judges whether image recognition information is suitable. An update process is not performed 
when unsuitable. In the suitable case, LastSensorPos is reduced and set up from the positional information 
according correction value to image recognition. 

[0075] In addition, in such a case, this invention is not restricted although it thought that the input from the 
position- sensor information processing section 403 to the board image recognition section 502 could be 
disregarded here. When highly precise information is acquired from a position sensor 104, the board image 
recognition section 502 can also update correction value by the same approach as the position-sensor 
information processing section. Moreover, you may change how many values are amended mutually by the 
reliability of each value. For example, when the difference of reliability is very large, the correction value of the 
positioning means of a low side is updated so that the output of a high side may be outputted almost as it is, but 
when there is no difference not much, it is condition of updating correction value so that it may amend little by 
little and may suit mutually. 

[0076] According to this operation gestalt, the positional information of the player view of always high 
reliability can be acquired with the continuous value by updating correction value. Therefore, the displeasure by 
CG drawing location shifting suddenly can be prevented. 

[0077] (5th operation gestalt) Although it stated above when there were some which combined the gyroscope 
sensor and the acceleration sensor as a method of a position sensor, a gyroscope sensor detects only attitude 
information. If the position is beforehand proofread when using such an attitude sensor, it can use as a position 
sensor. However, proofreading is unnecessary when it has in coincidence the positioning means which consists 
of a camera 501 and the board image recognition section 502. 

[0078] Then, although positional information is fundamentally calculated by the image processing, only posture 
data can be complemented with the value of an attitude sensor when the positional information by the image 
processing is inaccurate, if ****** by change of the visual field by position change of HMD 103 and posture 
change of HMD 103 is compared the case of a board game or a card game, since it will be thought that latter one 
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is dominant, it is also greatly significant to complement only attitude information. Then, the attitude sensor 
other than a camera 501 is fixed to HMD 103. 

[0079] This is the example of the image experience system of the 5th operation gestalt, and shows the example 
of a configuration to drawing 1 1 . 

[0080] Only the configuration of the player positioning means 302 has changed compared with drawing 6 . It is 
maximum ************ uq3 instead of the position sensor 104 instead of the attitude sensor information 
processing section 1 102 and maximum ************ 601 instead of an attitude sensor 1 101 and the position- 
sensor information processing section 403. 

[0081] The flow of fundamental processing is not different from the case of the 3rd operation gestalt. The 
attitude sensor information processing section 1 102 processes the output data of an attitude sensor 1101, and 
attitude information is outputted. In addition to attitude information, position information was also outputted in 
the position-sensor information processing section 403, but the points differ. 

[0082] An UML activity Fig. is shown for processing of maximum ************ 1 103 in drawing 12 . 
[0083] It first waits to come [ of the attitude sensor information from the attitude sensor information processing 
section 1 102, and the image recognition information from the board image recognition section 502 ]. If both 
data are assembled, it will judge whether image recognition information is suitable. 
[0084] If it is judged that it is suitable, only the position information of them will be set as object variable 
LastlPPos. And image recognition information is returned and processing is ended. 

[0085] Although attitude sensor information will be used as attitude information supposing it is judged that it is 
unsuitable, LastlPPos previously set up as insufficient position information is used. The positional information 
acquired combining both data is returned, and processing is ended. 

[0086] In addition, although the value of the board image recognition section assumes that reliability is high 
rather than the value of the attitude information processing section about attitude information here, it can 
respond to mutual reliability, and maximum ************ H03 can calculate and determine attitude 
information from both value. 

[0087] According to this operation gestalt, by the maximum ************ ? since the posture of a most reliable 
player view is determined based on each output value, even when one output is unsuitable, the posture of a most 
reliable player view can be acquired with the reliability of each output value. 

[0088] Moreover, proofreading of the location of a player view which was required only in the case of the 
attitude sensor is unnecessary. Moreover, since an attitude sensor is used instead of a position sensor, it can 
constitute cheaply. 

[0089] (6th operation gestalt) A value swings as the attitude information acquired from the attitude sensor 
information processing section 1 102 was also explained above. Then, this problem is conquered by the same 
approach as the 4th operation gestalt. 

[0090] Drawing 13 is drawing showing the example of the image experience structure of a system of the 6th 
operation gestalt. Although the configuration has not changed compared with drawing 1 1 , it differs in that the 
output of the board image recognition section 502 is inputted also into the attitude sensor information 
processing section 1 102 in addition to maximum ************ H03. However, the value passed to the attitude 
sensor information processing section 1 1 02 is only the attitude information of the positional information. 
[0091] The UML activity Fig. showing processing of the attitude sensor information processing section 1 102 is 
shown in drawing 14 . 

[0092] It usually passes along the information from an attitude sensor 1 101, it is processed, and attitude 
information is calculated. The value is evacuated to Variable LastSensorDir. Variable LastSensorDir is an 
object variable and is referred to also from another thread described just in the back. 

[0093] Then, it evacuates to the calculated attitude information temporarily by making into a return value the 
value which applied correction value. In addition, correction value is also an object variable and a value is set 
up by another thread described just in the back. Although a return value is a variable, this is only temporarily 
used, in order to perform exclusive activation. And finally a return value is returned as an output of the attitude 
sensor information processing section 1 102. 

[0094] Now, although it is necessary to calculate above-mentioned correction value, this is updated at any time, 
when there is an output from the board image recognition section 502. 

[0095] First, it judges whether image recognition information is suitable. An update process is not performed 
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when unsuitable. In the suitable case, LastSensorDir is reduced and set up from the attitude information 
according correction value to image recognition. 

[0096] In addition, in such a case, this invention is not restricted although it thought that the input from the 
attitude sensor information processing section 1 102 to the board image recognition section 502 could be 
disregarded here. When highly precise information is acquired from an attitude sensor 1 101 , the board image 
recognition section 502 can also update correction value by the same approach as the attitude sensor 
information processing section. Moreover, you may change how many values are amended mutually by the 
reliability of each value. For example, when the difference of reliability is very large, the correction value of the 
part in connection with the attitude determination within the attitude determination means of a low side or a 
positioning means is updated so that the output of a high side may be outputted almost as it is, but when there is 
no difference not much, it is condition of updating correction value so that it may amend little by little and may 
suit mutually. 

[0097] According to this operation gestalt, positional information including the attitude information of the 
player view of high reliability can be acquired with the continuous value. 

[0098] (7th operation gestalt) Now, it is as having already stated to advance a game by some fields 1 being on the 

board 101 which is the place used with a board game, and a player f s putting a coma on each field, removing, 

and moving between fields. When the game Management Department 401 grasps the scene situation or advance 

situation of a game, CG generation section 405 can generate CG suitable for the scene situation or advance 

situation of a game, and is sensed as a game with a more high touch of reality for a player. 

[0099] then, the coma which a player operates ~ being related - "which coma" and "which field" - "~ it was 

placed — it /removed — " — a coma actuation recognition means to recognize is given. 

[0100] In addition, a game is advanced not only using a coma but using other items, and you may make it 

recognize actuation of the item. 

[0101] Drawing 15 is the example of the image experience structure of a system of the 7th operation gestalt. 
[0102] The coma actuation recognition means 1501 consists of a special mark of the bar code attached to a 
coma, and special mark recognition means 1502, such as a bar code reader which recognizes it. However, since 
a special mark is attached to a coma, it is not illustrating to drawing 15 . 

[0103] In addition, since a special mark is only for identifying a coma, the system called RFID not only using 
the mark by general printing but IC chip etc. or a similar system can also be used. 

[0104] The special mark recognition means 1502 may be installed according to an individual for every field on 
a board 101, and or it is plurality, all fields may be summarized and it may be provided with one set. 
[0105] After the data from the special mark recognition means 1502 are passed to the special mark recognition 
section 1503, they are passed to the coma actuation recognition section 1504. 

[0106] In the special mark recognition section 1503, the information from the special mark recognition means 
1502 is interpreted, and it changes into the data format which the coma actuation recognition section 1504 
needs. Since "which field" can be judged by the special mark recognition section 1503 which takes out an 
output supposing every one special mark recognition section 1 503 corresponds to each field, it is not necessary 
to output the information but, and when the one special mark recognition section 1 503 provides two or more 
fields, the information of "which field" is also outputted about the field of the range currently provided. 
Moreover, when the inputs from the special mark recognition means 1502 are ten digits, for example, it carries 
out using a conversion table etc. and changes into the information of "which coma." 

[0107] the coma actuation recognition section 1504 — "which coma" and "which field" — "— it was placed — 
it /removed — " — it recognizes and a recognition result is passed to the game Management Department 401 as a 
result of the coma actuation recognition means 1501. 

[0108] At the game Management Department 401 , a game is advanced based on the recognition result from the 
coma actuation recognition means 1501. In an actual game advance, the information that "which field" 
migration "which coma" and "from which field" was carried out may be required. When a coma combines the 
information "it removed from Field j", and the information "it was put on the coma i field k", the game 
Management Department 401 judges this. In this case, if it becomes and the coma put on Field j will be i, it will 
mean, "Coma i moved to Field k from Field j." The cognition by the game Management Department managing 
and referring to hysteresis is possible for saying [ that the coma put on Field j was i ]. 

[0109] Drawing 16 shows processing of the coma actuation recognition section in the activity Fig. of UML. 
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[01 10] The special mark recognition means 1502 is installed in each one field of every on a board 101, and the 
special mark recognition means i supports Field i. Moreover, the special mark recognition section 1503 shall 
return the special special mark identifier Nothing, when Coma j is placed and j is removed as a special mark 
identifier. 

[01 1 1] If waiting and the special mark identifier j are Nothing(s), it will suppose the input from the special mark 
recognition section, "A coma was removed from Field i", otherwise, it will be considered as the result of the 
coma actuation recognition means 1501 as "Coma j was put on Field i." 

[0112] According to this operation gestalt, it can be advanced based on the actuation in which the player 
actually ran the game. And since CG suitable for the scene situation or advance situation of a game is generable, 
for a player, it is effective in sensing as a game with a more high touch of reality. 

[0113] (8th operation gestalt) A bar code can be used as a special mark identifier. This corresponds to claim 10. 
[0114] The bar code is widely used in fields, such as PD, and there is the cheap description of ** which 
acquisition tends to carry out that highly precise recognition and recognition are stable. Especially in the case of 
a card game, a bar code can be printed to coincidence at the time of printing of a card. Moreover, it is effective 
in the ability to give a special mark, without affecting a design, if an invisible bar code is used. 
[0115] (9th operation gestalt) A RFID system can be used as a special mark recognition means. RFID is the 
technique called Radio Frequency Identification, and is a non-contact automatic discernment technique by radio 
frequency. 

[01 16] What is called a tag or a transponder is attached in a body, and ID of a tag proper is read by the reader. 
Generally, a tag consists of the semi-conductors and antennas with which the single chip of a transceiver circuit, 
the control circuit memory, etc. was carried out. Although a reader discharges a question electric wave, in order 
to use this question electric wave as electrical energy, to a tag, a cell is unnecessary. A tag discharges ID 
beforehand stored in memory to a question electric wave. A reader reads this ID and identifies a body. 
[01 17] A RFID system is the technique currently widely used with the ID card etc., and has the ease of 
acquisition, the highly precise recognition recognized and stabilized, and the description of being cheap. 
Moreover, if it dedicates to the interior of a coma, it is effective in the ability to perform recognition of a coma, 
without affecting an appearance at all. Furthermore, it is effective in the shape of surface type not being a flat 
surface, and a degree of freedom appearing in the design of a coma or the board itself, since an obstruction may 
be between a tag and a reader as long as it is the body of a nonmetal. 

[01 18] (10th operation gestalt) It is possible to recognize "which coma" also by carrying out image recognition 
processing of the image with which ** was also obtained using the camera not using the special mark 
recognition means. For example, if [ like chess for a pattern that it was drawn on the card face when it was a 
card game ], a pattern that the configuration of a coma was further drawn on the configuration of a coma and 
there depending on the game is used for coincidence, and a coma is recognized. 

[01 19] Although it takes for an example recognizing a pattern that it was drawn on the square card face here, 
even when using for coincidence a pattern that it was drawn on the configuration of a coma, and there and 
recognizing it even if this invention recognizes the configuration of a coma and, it can be applied. 
[0120] Drawing 17 is the example of the image experience structure of a system of the 10th operation gestalt. 
Compared with drawing 1 5 , only the configurations of the coma actuation recognition means 1 501 differ, the 
special mark recognition means 1502 supports to the coma recognition camera 1701, and the special mark 
recognition section 1503 supports the coma image recognition section 1702. In addition, the coma actuation 
recognition section 1 504 is the same. 

[0121] Although the example which recognizes the thing of two patterns shown in drawing 18 R> 8 as a pattern 
that it was drawn on a coma is shown here, if still more complicated processing is used, it is also possible to 
recognize complicated various patterns like comics or a photograph. 

[0122] Drawing 20 shows processing of the coma image recognition section 1702 in an UML activity Fig. 
[0123] There are two steps in recognition, first, it detects, a frame continues, and it is detection of a pattern. 
When a frame is undetectable, it judges that there is no card and Nothing is returned as a coma identifier. 
Although especially detection of a frame is not illustrated, a straight line is detected using the Hough conversion 
etc. and there is the approach of judging from those physical relationship to be a frame etc. 
[0124] When a frame is able to be detected, a pattern is detected next. As shown in drawing 19 , the color of 
each field is detected by dividing into four for within the limit. There is some technique also in color detection. 
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For example, in detection of only white and black, the average of the lightness of the field which it is going to 
detect is calculated only using lightness information, and if it becomes below the constant value TB with the 
value and will become black and more than a certain constant value TW, it can judge with white. 
[0125] As shown in drawing 19 , the number of 1 to 4 was attached to each field. If it is monochrome black and 
white or monochrome black and white, 1 will be returned as a coma identifier, and the result of having arranged 
the color to fields 1-4 in order will return 2, if it is deep black white white, white monochrome black, 
monochrome black and white, or monochrome black and white. If it is except it, it will be the card which is not 
assumed, or will be incorrect recognition of a frame etc., and this will return Nothing as a thing without a card. 
In addition, when carrying out image recognition continuously, the same result comes out continuously. 
[0126] Moreover, the condition of having said that the midst which has placed the card, or the midst which it is 
going to remove was as remove **** [ and ] may be outputted at random. [ that a card is placed ] Although the 
device that the same continuous output outputs when it controls or the same condition continues beyond fixed 
time amount is needed in a game from which these pose a problem, it does not explain here. 
[0127] (1 1th operation gestalt) Asa**** camera for recognition of a coma, the camera 501 fixed to HMD 103 
may be used. Since the camera fixed to HMD of a player is used as a camera used with a coma actuation 
recognition means, equipment can be simplified. 

[0128] By the board image recognition section 502, if a board 101 can be recognized, the field prepared on the 
board 101 can be judged. Then, it can consider as a coma actuation recognition means by recognizing a coma in 
the field. 

[0129] Drawing 2 1 is the example of the image experience structure of a system of the 1 1th operation gestalt. 
The coma actuation recognition means 1501 consists of the board top coma image recognition section 2101 and 
the coma actuation recognition section 1504, the image data to recognize is inputted from a camera 501, and the 
recognition information on a board 101 is inputted from the board image recognition section 502. Although the 
output of the board image recognition section 502 is the location of a player view, if the location of a player 
view is known, the location of the board on an image is easily calculable from the information. 
[0130] Drawing 22 shows processing of the board top coma image recognition section in an UML activity Fig. 
The image input from a camera 501 is obtained, and the location of the HMD view corresponding to the image 
is continuously obtained from the board image recognition section 502. 

[0131] From the location of a HMD view, the location of the board 101 on an input image is calculated first. 
Subsequently, from the positional information of each field on the board decided beforehand, the location of 
each field on an input image is calculated. 

[0132] A coma on each field is recognized by cutting down the image of the location, if the location of each 
field is known, and performing image recognition. Under the present circumstances, since the information on a 
posture is also included in the positional information of each field, precision can also be raised by also using 
this information and performing image recognition. 

[0133] (12th operation gestalt) When recognizing a coma on a board using a camera 501, the distance from a 
camera 501 to a coma, the posture of a coma over a camera 501, etc. pose a problem, and in order that the 
number of pixels which a coma on an image has decreases, and recognition may become difficult or may amend 
deformation of an image, the configuration of the image recognition section may become complicated. 
[0134] Then, in case it recognizes "which coma it is", a coma is recognized by bringing a coma to the specific 
location defined beforehand from a camera 501. 

[0135] For example, it is condition of bringing a coma to the place of 30cm of views. If it is a card as shown in 
drawing 18 , when a frame comes to the specific location on an image, it can judge with having held up the card 
and will recognize there. 

[0136] What is necessary is to pursue a coma until "it places where" is placed on a board, to simplify the image 
recognition section in the 10th operation gestalt, and just to make it "something was placed" and recognize, if a 
coma can be recognized. It relates [ "it removed" and ] and is the same. 

[0137] In case "which coma" is recognized, a recognition rate can be gathered to the specific location of a 
specific camera by arranging a coma. Furthermore, the configuration of the recognition section can be 
simplified. 

[0138] Drawing 23 is the example of the image experience structure of a system of the 12th operation gestalt. 
Compared with drawing 2 1 , only the configurations of the coma actuation recognition means 1501 differ. 

http://www4.ipdl.ncipi.go.jp/cgi-bin/tran_web_cgi_ejje 11/2/2005 



JP,2003-103045,A [DETAILED DESCRIPTION] 



Page 11 of 13 



Although the thing of a publication with the 1 1th operation gestalt is sufficient as the board top coma image 
recognition section 2101, since a coma should just judge [ "it was placed", "it having removed", and ], it may be 
simplified further. The coma image recognition section is the same as that of what was indicated with the 10th 
operation gestalt. The coma actuation recognition section 2301 differs from the coma actuation recognition 
section 1 504 somewhat, and obtains an input from both board top coma image recognition section 2101 and 
coma image recognition section 1702. 

[0139] An UML activity Fig. shows processing of the coma actuation recognition section 2301 to drawing 24 
and drawing 25 . the time of there being information "Coma j has been recognized" from the coma image 
recognition section 1702 — drawing 24 — from the board top coma image recognition section 2101 — "— a coma 
was put on Field i — it /removed — " — ** — the time of there being information to say is shown in drawing 25 . 
[0140] When "Coma j" is recorded on the object variable when there is information "Coma j has been 
recognized" from the coma image recognition section 1702, and the information "a coma was put on which 
field" is in behind, it uses as information of "which coma." 

[0141] When there is information "a coma was put on Field i" from the board top coma image recognition 
section 2101, the "coma j" recorded previously is read from an object variable, and the result "Coma j was put 
on Field i" is returned. 

[0142] Moreover, when there is information "a coma was removed from Field i" from "the board top coma 
image recognition section 2101 ", the result "a coma was removed from Field i" is returned as it is. 
[0143] In addition, the camera of dedication like the paintings-and-calligraphic- works camera prepared 
separately instead of the camera 501 fixed to HMD103 in the input to the coma image recognition section 1702 
may be used. Moreover, even if it transposes the camera of dedication, and the combination of the coma image 
recognition section 1702 to the special mark recognition means 1502 and the special mark recognition section 
1503 which were prepared separately, there is same effectiveness. 

[0144] (13th operation gestalt) Although a coma is recognized by holding up a coma in front of a camera 501, 
for a player, it is rather unclear to which location it should bring, moreover — if it is easy to use a player as an 
image experience system side as possible — the spatial recognition range — large ~ not taking — it does not 
obtain, but if it does so, complicated and an advancement of the recognition section, and decline in a 
recognition rate will be caused. 

[0145] What is necessary is just to be able to hold up a coma to the recognition range narrow on the space target 
set beforehand, without a player straying. When it thinks that a player will make it recognize a coma, a guide is 
displayed on the display 303 of HMD101, and a player should just hold up a coma so that it may double with 
the guide. 

[0146] The example of the image experience structure of a system of the 13th operation gestalt is shown in 
drawing 26 . Although the configuration has not changed compared with drawing 23 , the coma image 
recognition section 1702 is changed into coma image recognition and the guide display directions section 2601, 
and only the parts by which information is outputted to CG generation section 405 from here differ. 
[0147] Although coma image recognition and the guide display directions section 2601 are the almost same 
configurations as the coma image recognition section 1702, when the reliability over a recognition result is the 
following to some extent, it differs in that guide display directions are issued. The difference in the output of the 
coma image recognition section 1702, and the coma image recognition and the guide display directions section 
2601 is shown in drawing 27 . 

[0148] Supposing it uses the same recognition engine as the coma image recognition section 1702, it will judge 
that it has recognized if it was beyond a certain value Th with the reliability of recognition higher than the coma 
image recognition section 1702, and a recognition result will be outputted. Since this threshold Th is high, it 
becomes easy to realize a high recognition rate. 

[0149] Although it judges that un-recognizing, i.e., a coma, is not held up in the case of reliability lower than a 
certain value Tl, in the case of the coma image recognition section 1702, this value Tl is the same as the value 
Th in the case of the ability to recognize. However, in the case of coma image recognition and the guide display 
directions section 2601 , it has set up low. In the case of the reliability "is also recognized", guide display 
directions will be issued. [ this "recognition" or ] 

[0150] An UML activity Fig. shows processing of coma image recognition and the guide display directions 
section 2601 to drawing 28 . In addition, when reliability is made or less [ 0 or more ] into one, there is relation 
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of 0<Tl<=Th<L Coma image recognition processing is performed, and if the reliability over the recognition 
result is lower than Tl, nothing will be performed as un-recognizing. If reliability is higher than Th, a 
recognition result will be passed to the coma actuation recognition section 2301 noting that it is able to 
recognize. When it is not which [ the ], either, guide display directions are issued to CG generation section 405. 
A guide display is like drawing 29 . 

[0151] Since according to this operation gestalt the guide make a coma easy to arrange is constituted from CG 
and a synthetic indication is given at HMD of a player in case a coma is arranged in the specific location of a 
specific camera, it is easy for a player to hold up a coma to the suitable location on space. 
[0152] (14th operation gestalt) The game performed with two or more players needs to share the occurrence on 
a board 101 with all players also including a display. This is realizable by sharing the one game Management 
Department 401 between all players logically. Physically, it may be arranged on two or more game consoles or 
PC like a distributed database by one set also of the specific exclusive game console which contains other 
components also as one set of dedication PC. 

[0153] That is, for each player, the game console or PCI 02 has taken a configuration like drawing 4 , and the 
game Management Department 401 is also reflecting the result of the actuation which other players performed. 
[0154] Drawing 30 shows the example of the image experience structure of a system of the 14th operation 
gestalt. 

[0155] One set each of a game console and PC 102 are assigned to one player, and it connects mutually in the 
network. The information which flows a network is the information for taking the synchronization of the 
contents of each game Management Department 401 in each game console or PC 102. 

[0156] Moreover, although the coma actuation recognition means is arranged at each game console or PCI 02, 
recognition will be performed for the same coma from two or more views in this case. In such a case, through a 
network, information is exchanged mutually and the recognition result of a side with more high reliability can 
also be used. 

[0157] Drawing 3 1 is also the example of the image experience structure of a system of the 14th operation 
gestalt. Game contents are in the game server on the Internet, and the game console or PC 102 of each player is 
connected through the Internet. 

[0158] The game Management Department 401 consists of the local game Management Department 3101 and a 
game server 3102, and the game server 3102 is arranged at the independent machine. The local game 
Management Department 3101 treats that it is related only to each player, and that feedback to each local player 
with which time delay poses a problem is the need. Moreover, relating to the contents of each game contents 
downloads data and a program from the game server 3102 through a network in the time of game initiation, or a 
game. 

[0159] According to this operation gestalt, two or more players can play on one board, the result of the 
combined control by two or more players can be expressed to each HMD as each player view, and the game 
pitched against each other with two or more players can be experienced. 

[0160] (Other operation gestalten) To the computer in the equipment which operates various kinds of devices so 
that the function of the gestalt of operation mentioned above may be realized and which was connected with 
these various devices like, or a system The program code of the software for realizing the function of the gestalt 
of said operation is supplied. What was carried out by operating said various devices according to the program 
in which the computer (CPU or MPU) of the system or equipment was stored is contained under the category of 
this invention. 

[0161] In this case, the function of the gestalt of operation which the program code of said software itself 
mentioned above will be realized, and the storage which stored the means for supplying that program code itself 
and its program code to a computer, for example, this program code, constitutes this invention. 
[0162] As a storage which stores this program code, for example, a floppy (trademark) disk, a hard disk, an 
optical disk, a magneto-optic disk, CD-ROM, a magnetic tape, the memory card of a non-volatile, ROM, etc. 
can be used. 

[0163] Moreover, by performing the program code with which the computer was supplied, also when the 
function of the above-mentioned operation gestalt is not only realized, but the function of the above-mentioned 
operation gestalt is realized in collaboration with OS (operating system) to which the program code is working 
in a computer, or other application software, it cannot be overemphasized that this program code is contained in 
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the operation gestalt of this invention. 

[0164] Furthermore, also when the function of the operation gestalt which performed a part or all of processing 
that CPU with which the functional add-in board and a functional storing unit are equipped based on directions 
of the program code is actual, and mentioned above by the processing is realized after the supplied program 
code is stored in the memory with which the functional expansion unit connected to the functional add-in board 
and the computer of a computer is equipped, it cannot be overemphasized that it is contained in this invention. 
[0165] 

[Effect of the Invention] The advance situation of a game can be made further easy according to this invention, 
to be alike in the fun of an actual board game, in addition for presence to increase, and to grasp. 
[0166] And as compared with the game using the conventional MR technique, installation is easy and it can 
respond also to modification of the contents of a game comparatively flexibly. 
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* NOTICES * 

JPO and NCIPI are not responsible for any 
damages caus d by the use of this translation. 

1 .This document has been translated by computer. So the translation may not reflect the original precisely. 
2.**** shows the word which can not be translated. 
3. In the drawings, any words are not translated. 
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